Video games may use controllers that generate rich data streams. For example, a depth camera may be used to generate a control signal for a video game. A depth camera generates a three-dimensional image of the space viewed by the camera. Other rich data streams include video and audio data. Player movement within the three-dimensional image may be detected and interpreted, perhaps in conjunction with other input data, to manipulate a video game.